Selectively controllable heads-up display system

ABSTRACT

Systems and methods are disclosed for displaying data on a head&#39;s-up display screen. Multiple forms of data can be selectively displayed on a semi-transparent screen mounted in the user&#39;s normal field of view. The screen can either be mounted on the user&#39;s head, or mounted on a moveable implement and positioned in front of the user. A user interface is displayed on the screen including a moveable cursor and a menu of computer control icons. An eye-tracking system is mounted proximate the user and is employed to control movement of the cursor. By moving and focusing his or her eyes on a specific icon, the user controls the cursor to move to select the icon. When an icon is selected, a command computer is controlled to acquire and display data on the screen. The data is typically superimposed over the user&#39;s normal field of view.

FIELD OF THE INVENTION

The inventions relate to electronic display systems and control systemstherefor. More particularly the inventions relate to selectivelyoperable heads-up display systems for presenting information and/orimage(s) to the user. In its preferred form, the heads-up display isconfigured for use by medical technicians or personnel, such as surgeonsperforming an operation.

BACKGROUND OF THE INVENTION

A heads-up display is generally defined as an electronically generateddisplay containing information or data that is superimposed on anobserver's normal field of view. As explained in greater detail below,heads-up display (“HUD”) systems have been used in various applications.One such application is for use by pilots of aircraft. In the typicalaircraft HUD system, a semi-transparent display screen is locatedgenerally in front of the eyes of the pilot (i.e. a screen mounted onthe pilot's head or helmet, or in the view of the aircraft windshield).Such a system enables a pilot to concentrate on the difficult tasksassociated with flying the aircraft, without diverting his attention toscan or examine a wide array of instruments.

It is also well known that medical technicians or personnel, such assurgeons, must keep track of many different types of information duringan operation. For example, a surgeon must carefully view or monitor thephysical surgery while simultaneously monitoring a patient's condition(e.g., blood pressure, heart rate, pulse, etc.). In addition, dependingon the procedure, the surgeon must also monitor the status and settingsof surgical equipment and tools. Although the additional information isnecessary and important, monitoring such information often diverts thesurgeon from the immediate task at hand.

Some surgical operations require that the surgeon divert his eyes toview a video monitor, for example, when performing highly complex laseror internal surgery conducted through scopes. See U.S. Pat. No.5,222,477, which discloses an endoscope or borescope stereo viewingsystem. In addition, the surgeon may from time to time need to refer todata, such as defined by a patient's recorded or written history, or topreviously taken x-rays or other computer generated images (e.g., CAT,NMR, 3D, etc.). For example, U.S. Pat. No. 5,452,416 discloses anautomated system and a method for organizing, presenting, andmanipulating medical images for viewing by physicians. See also U.S.Pat. Nos. 5,251,127 and 5,305,203, which disclose a computer-aidedsurgery apparatus that positions surgical tools during surgery orexaminations. In each of the above-described systems, in order to viewthe displayed information, the surgeon must divert his or her eyes to aremote monitor.

Thus, surgeons use many different types of displays and must continuallymonitor many different sources of information. However, as more toolsand sources of data become available to surgeons for use duringoperations, more opportunities for distraction arise. It is difficultfor a surgeon to focus on his or her conduct during a surgical procedurewhile also continually shifting focus away from the patient to othermonitors or indicators. Therefore, a need exists for conveniently,efficiently and accurately displaying to a surgeon various types andsources of information, views, and images of a patient undergoing acritical medical procedure. As explained in greater detail below, priorattempts in the medical field to fulfill that need have beenunsatisfactory.

For example, video signal sources have been adapted to scan views orimages for different types of medical uses and applications. U.S. Pat.No. 4,737,972 to Schoolman (“Schoolman I”) discloses a head-mounteddevice that provides stereoscopic x-ray images. Furthermore, U.S. Pat.No. 4,651,201 to Schoolman (“Schoolman II”) discloses an endoscope thatprovides stereoscopic images of the patient on a display. Both SchoolmanI and Schoolman II allows for the selective transmission of other videodata to the display. However, Schoolman I and Schoolman II do not use a“see through” display that allows the surgeon to monitor both theenvironment around him and the video image. If the surgeon wishes tomonitor or view the real-world environment, as opposed to the displayedgraphics, the head-mounted display must be removed.

Efforts have also been made to use head-mounted displays in augmentedreality simulations for medical applications wherein a desired image orthree-dimensional model is superimposed on the real scene of a patient.For example, it was reported that a research effort in the Department ofComputer Science at the University of North Carolina has attempted todevelop a see-through head-mounted display that superimposed acomputer-generated three-dimensional image of the internal features of asubject over the real-life view of the subject. Information describingthose research efforts may be found on the World Wide Web in a documentmaintained by Jannick Rolland at the site on the World Wide Web Pages ofthe NSF/ARPA Science and Technology Center for Computer Graphics andScientific Visualization at the University of North Carolina, ChapelHill (http://www.cs.unc.edu/˜rolland, cited February, 1996, copies ofwhich are included in the information disclosure statement that has beenfiled concurrently with this application). That World Wide Web site inturn referenced the following publication: A. R. Kancheral, et al., “ANovel Virtual Reality Tool for Teaching 3D Anatomy,” Proc. CVR Med '95(1995). Other research efforts at the University of North Carolinaattempted to use a video see-through head-mounted display and ahigh-performance computer graphics engine to superimpose ultrasoundimages over the real view of the subject, thereby allowing a user to“see within” the subject. A set of trackers captured the motion of thebody part with respect to the field of view of the user, and a computerupdated the position of the body part in real time. The computerattempted to correlate the “tracked” position of the body with thethree-dimensional model and to display the model on the heads-up displayin a manner that gave the appearance of “x-ray vision.”

In the above-described University of North Carolina research efforts,the focus was primarily to help teach students by superimposing a singlecomputer-generated image over a moving, real-life, image of a subject.However, as explained in the associated literature, the “tracking”requirements made the research effort quite complicated, and the resultsappeared less than satisfactory. Moreover, such a teaching system is notapplicable to the real-world environment of a surgeon, where the patientis not moved (and “tracking” is unnecessary), and where the surgeonneeds or desires other information to be made readily available forviewing.

Still another research program associated with the University of NorthCarolina is described in Fuchs, et al., “Virtual Space Teleconferencingusing a Sea of Cameras,” Proceedings of the First InternationalSymposium on Medical Robotics and Computer Assisted Surgery (Pittsburgh,Pa, Sep. 22-24, 1994). That article describes research efforts thatattempted to use a multitude of stationary cameras to acquire bothphotometric and depth data. The acquired data was purportedly used toconstruct a remote site in accordance with the head position andorientation of a local participant. According to the article, eachparticipant wears a head-mounted display to look around a remoteenvironment having surface geometries that are continuously sensed by amultitude of video cameras mounted along the walls and ceiling, fromwhich cameras depth maps are extracted through cross-correlation stereotechniques. Views acquired from several cameras are then displayed on ahead-mounted display with an integrated tracking system to provideimages of the remote environment. The explained purpose of the effortwas to duplicate, at a remote location, a three-dimensional virtualreality environment of a medical room. However, the article does notdisclose the use of see-through displays providing a surgeon with theability to select and display additional forms of data, or tosuperimpose data over a real-life view of the patient or surgical site.

Another type of head-mounted display is described in Yoshida, et al.,“Optical Design and Analysis of a Head-Mounted Display with aHigh-Resolution Insert,” Proc. SPIE 2537 (1995). That article describesyet another research program associated with the University of NorthCarolina in which a small area of a high-resolution image is inserted ona large field of a low resolution image displayed on a head-mountedscreen. The system is described as using eye-tracking information todynamically place the high resolution insert at the user's gaze point.The system purports to provide the user with both high-resolutionimagery and a large field of view. In essence, using eye-trackingelectronics, the “inserted image” corresponding to the user's gaze pointis converted from low resolution to high-resolution. However, as above,the user can not select additional or alternative forms of data ordifferent images to be superimposed over the primary image on thehead-mounted display.

Thus, few head-mounted displays have been developed for the medicalindustry, and all those described above have had limited purpose andutility. On the other hand, and as discussed briefly above, a widevariety of head-mounted devices are commonly used in militaryapplications. As mentioned, aircraft pilots, tank commanders, weaponoperators and foot soldiers have all used head-mounted displays todisplay various forms of weapon or image information along with otherdata defining the real-world environment of the person wearing thedisplay. For examples of such systems, see the following U.S. Pat. Nos.4,028,725; 5,281,960; 5,000,544; 5,227,769; 4,994,794; 5,341,242;4,878,046; 3,940,204; 3,923,370; 4,884,137; 4,915,487; and 4,575,722.Likewise, helmet or head-mounted displays have also been used formotorcycle riders. U.S. Pat. No. 4,988,976 discloses a motorcycle helmetthat displays data or information such as speed, time, rpm's, fuel, oil,etc. on the transparent visor (i.e. vacuum fluorescent display) of therider. Head-mounted displays that are worn in front of the user's eyesor worn as eye spectacles also exist. For example, see the followingU.S. Pat. Nos. 5,129,716; 5,151,722; 5,003,300; 5,162,828; 5,331,333;5,281,957; 5,334,991; 5,450,596 and 5,392,158.

The field of virtual reality also has driven advances in the use ofvarious types of head-mounted displays. For example, see the followingU.S. Pat. Nos. 4,636,866; 5,321,416; 5,347,400; 5,348,477; 5,406,415;5,414,544; 5,416,876; 5,436,765; 5,479,224; 5,473,365; D363,279;5,485,172; 5,483,307; 5,130,794. See also the publication How VirtualReality Works, by J. Eddings (Ziff-Davis Press, Emeryville, Calif.,1994), and the site maintained by Rolland (referenced above) relating totelepresence systems and augmented reality.

Advances have also been made in the area of heads-up displays or screensthat are not attached to or worn by the user. Most commonly, suchsystems are employed in automotive or military environments, to providevehicle performance, weapon status, and other data for the driver orpilot. For examples of such systems, see the following U.S. Pat. Nos.5,278,696; 4,652,870; 4,711,512; 4,729,634; 4,799,765; 4,927,234;4,973,139; 4,988,976; 4,740,780; 4,787,711; 4,740,780; 4,831,366;5,005,009; 5,037,182; 5,231,379; 4,824,228; 4,763,990; 4,669,810;4,688,879; 4,818,048; 4,930,847; 4,932,731; 5,198,895; 5,210,624;5,214,413; 5,302,964; 4,725,125; 4,188,090; 5,028,119 and 4,769,633.

Numerous advances have occurred in the specific forms of, and materialsused in, heads-up display systems. See, for example, U.S. Pat. Nos.4,987,410 and 4,961,625 (use of Liquid Crystal Displays (LCDs)); U.S.Pat. Nos. 5,108,479 and 5,066,525 (laminating glass plates or panels);and U.S. Pat. No. 5,457,356 (making a flat panel head-mounted display).

Also pertinent to this invention is the field of eye-tracking to controlvarious computer or imaging functions. Various systems unrelated to themedical field have used eye-tracking for controlling a field of view.For example, see U.S. Pat. No. 4,028,725, which discloses an eye andhead tracking system that controls a beam-splitter and retains thehigh-resolution part of the image in the field of view. The eye-trackingis carried out by infrared detection (i.e. see U.S. Pat. No. 3,724,932).See also U.S. Pat. Nos. 5,287,437; 4,439,755; 4,349,815; 4,437,113;4,028,725 (referenced earlier) and the article “Optical Design andAnalysis of a Head-Mounted Display with a High-Resolution Insert,”referenced above, particularly at footnote 19, which refers to theEye-tracking Systems Handbook, Applied Science Laboratories, WalthamMass. (1992).

Finally, video recording systems for recording scenery and heads updisplays have been taught by the prior art. U.S. Pat. No. 5,241,391 toDodds (“Dodds”) discloses a video camera system that records sceneconditions and heads-up displays.

Notwithstanding the large number of articles and patents issued in thearea of heads-up or head-mounted displays, there has been no suchdisplay that is designed for the special needs of individuals performingdetailed but critical tasks on relatively stationary subjects. Such asystem would be extremely useful to personnel working in the fields ofmedicine, forensics, and micro-electronics.

Presently, there is a need for a selectively operable, head-mounted,see-through viewing display system for presenting desired informationand/or images to a user, while at the same time allowing the user toview the real-world environment in which he or she operates. There is afurther need to provide a convenient selectable viewing system that canbe easily controlled by an eye-tracking cursor and speech recognition tocontrol different images or displays on a videomonitor or to control afield of view, while keeping the user's hands free to conduct precisionoperations.

SUMMARY OF THE INVENTION

Accordingly, it is an object of this invention to provide an improvedheads-up display system.

It is another object of the invention to provide a “hands-free” heads-updisplay system that is useful to individuals performing detailedprocedures, such as those working in the fields of medicine, forensics,micro-electronics, biotech, etc.

It is another object of the invention to provide an improvedhead-mounted display that allows the user to view both the subject andselected data.

It is another object of the invention to provide an improved heads-updisplay that includes a user-friendly interface to a command controlcomputer.

It is another object of this invention to provide an improved heads-updisplay that interfaces with a command control computer and includes aneye-tracking cursor to select menus to control computer performance andthe display of data.

It is another object of this invention to provide an improved heads-updisplay that interfaces with a command control computer and includes aspeech recognition circuit to control computer performance and displayof data.

It is another object of the invention to provide an improved heads-updisplay that can be positioned between a surgeon and a patient in thesurgeon's line of sight.

It is another object of the invention to provide an improved heads-updisplay that allows the user to view the subject while simultaneouslymonitoring the output from a number of different information sources,including imaging devices and remote or networked computer systems.

It is another object of the invention to provide an improved heads-updisplay that allows a medical technician to control medical imagingdevices to obtain images of select parts of a patient and to displaythose images on the heads-up display.

It is another object of the invention to provide an improved heads-updisplay that allows a user to control a computer to acquire and displaydata defining a subject's history while simultaneously viewing thesubject.

It is another object of the invention to provide an improved method forconducting surgery on a patient while simultaneously obtaining access toand conveniently displaying on a heads-up display a variety of types ofdata relating to the patient.

It is another object of the invention to provide an improved method ofcontrolling a heads-up display by employing a “point-and-click” typeuser interface and cursor controlled by tracking movement of the eye.

It is another object of the invention to provide an improved method ofcontrolling a heads-up display by employing speech recognition, bothalone and in combination with an eye-tracking cursor.

It is another object of the invention to provide an improved heads-updisplay system that allows the user to control tools or instruments in ahands-free manner.

It is another object of the invention to provide an improved heads-updisplay system that allows a surgeon to control surgical tools or otherinstruments in a hands-free manner.

It is another object of the invention to provide an improved heads-updisplay maintained in an eyepiece of a scope or instrument and that iscontrolled with integral eye-tracking and speech recognition systems.

The above and other objects are achieved in an improved, selectivelycontrollable system for presenting desired data on a head-mounted (or“heads-up”) display. The system includes a command computer processorfor receiving inputs that represent data and for controlling the displayof desired data. The computer communicates with and controls theheads-up display system, which is configured to display the desired datain a manner that is aligned in the user's field of view. The heads-updisplay includes a user interface incorporating “hands-free” menuselection to allow the user to control the display of various types ofdata. In its preferred form, the hands-free menu selection is carriedout using an eye-tracking cursor and a speech recognition computer topoint to and select specific menus and operations.

The above and other objects are also achieved in an user-controllableheads-up system for presenting medical data to a physician. The systemincludes a command control computer for receiving inputs definingmedical data and for controlling the display of that data on a head's-updisplay screen in the normal field of view of the physician. Theheads-up display provides the physician with a “user interface”including menus and associated operations that can be selected with aneye-tracking cursor. The system also includes a microphone and speakerso that a physician can communicate with other personnel and computersboth locally and remote from the cite. The command computer includes aspeech recognition processor to respond to spoken commands of thephysician. The command computer also communicates with and receives awide array of data from other computers networked therewith. Thephysician can select the specific data to be displayed on the screen. Inaddition, the physician can, with the eye-tracking cursor, controlvarious medical imaging devices.

The above and other objects are also achieved in a method of selectivelydisplaying multiple forms of data on a head-mounted display. Inaccordance with the method, a see-through computer display screen ismounted on a head piece that is worn by the user. A command computercontrols a user interface so that command icons or menus are displayedin a super-imposed manner on the see-through, head-mounted display,thereby allowing the user to see both the normal field of view and theuser interface. The user interface is provided with a “point-and-select”type of cursor. An eye-tracking system is integrated with the commandcontrol computer and the user interface to monitor the user's eyemovement and to correspondingly control movement of the cursor. The userselects various computer operations from menus contained in the userinterface by moving the eye-tracking cursor to selected menus or icons.By using the eye-tracking cursor to select various computer operations,the user can control the command computer to selectively display on thesee-through HUD screen numerous items of data or images, while stillseeing the normal field of view.

The preferred embodiments of the inventions are described below in theFigures and Detailed Description. Unless specifically noted, it is theintention of the inventors that the words and phrases in thespecification and claims be given the ordinary and accustomed meaning tothose of ordinary skill in the applicable art(s). If the inventorsintend any other meaning, they will specifically state that they areapplying a special meaning to a word or phrase.

Likewise, the use of the words “function” or “means” in the DetailedDescription is not intended to indicate a desire to invoke the specialprovisions of 35 U.S.C. Section 112, ¶ 6 to define his invention. Tothe, contrary, if the provisions of 35 U.S.C. Section 112, ¶ 6 aresought to be invoked to define the inventions, the claims willspecifically state the phrases “means for” or “step for” and a function,without also reciting in such phrases any structure, material or act insupport of the function. Even when the claims recite a “means for” or“step for” performing a function, if they also recite any structure,material or acts in support of that means of step, then the intention isnot to invoke the provisions of 35 U.S.C. Section 112, ¶ 6. Moreover,even if the inventors invoke the provisions of 35 U.S.C. Section 112, ¶6 to define the inventions, it is the intention that the inventions notbe limited only to the specific structure, material or acts that aredescribed in his preferred embodiments. Rather, if the claimsspecifically invoke the provisions of 35 U.S.C. Section 112, ¶ 6, it isnonetheless the intention to cover and include any and all structures,materials or acts that perform the claimed function, along with any andall known or later developed equivalent structures, materials or actsfor performing the claimed function.

As a primary example, the preferred embodiment of this invention isconfigured for use by a surgeon performing an operation on a patient.However, the invention is equally applicable to any environment in whichthe user is conducting precision or detailed procedures with his orhands on a relatively stationary subject, and where the user would findit advantageous to see data superimposed over the normal field of view.The potential applications are too numerous to mention, but wouldinclude forensics, microelectronics, biotechnology, chemistry, etc.Thus, even though the preferred embodiment refers to use by a surgeon,and to the acquisition and display of medical data, its applicability ismuch broader, and the claims should be interpreted accordingly.

Further, the description of the preferred embodiments make reference tostandard medical imaging devices that are used to generate images to bedisplayed on the heads-up display. The disclosure specificallyreferences several examples of such devices, including video cameras,x-ray devices, CAT and NMR scanners, etc. However, numerous othermedical imaging systems are well known to exist, and most likely,numerous improved imaging devices will be developed in the future. Thus,the present invention does not depend on the type of imaging device thatis implemented. The inventions described herein are not to be limited tothe specific scanning or imaging devices disclosed in the preferredembodiments, but rather, are intended to be used with any and allapplicable medical imaging devices. Likewise, the preferred embodimentsdepicted in the drawings show a single generic imaging device mounted ona manipulator arm. Numerous other tool and manipulator configurations,and multiple imaging devices, can be substituted for the single device.

Further, the specification in some places refers to several computers orcontrollers that perform various control operations. The specific formof computer is not important to the invention. In its preferred form,applicant divides several of the computing, control and analysisoperations into several cooperating computers or embedded systems.However, with appropriate programming well known to those of ordinaryskill in the art, the inventions can be implemented using a single, highpower computer. Thus, it is not the intention to limit the inventions toany particular form or any number of computers, or to any specificcomputer network arrangement.

Likewise, the detailed description below shows at least two embodimentsfor the display screen. The preferred embodiment discloses the displayscreen mounted on the head of the user. The second embodiment shows thedisplay screen positioned between the user and the subject, in a mannerthat is not mounted upon or supported by the head of the user.Additional embodiments also exist, and need not be disclosed. Forexample, the first embodiment can be modified for use in the eye-pieceof a scope of any form, such as used in micro-electronics, biotech,medicine, forensics, chemical research, etc.

Similarly, the specific arrangement of the icons and menus that appearon the HUD screen, and the associated operations performed by thoseicons and menu items, are a matter of choice for the specificapplication. Thus, the invention is not intended to be limited to thespecific arrangement and contents of the icons and menus shown anddescribed in the preferred embodiments. For example, the icons and menuitems for a selectively controllable heads-up display used by a dentistwould likely be different than the arrangement used for amicro-electronics engineer.

Further examples exist throughout the disclosure, and it is not theintention to exclude from the scope of the invention the use ofstructures, materials or acts that are not expressly identified in thespecification, but nonetheless are capable of performing a recitedfunction.

BRIEF DESCRIPTION OF THE FIGURES

The inventions of this application are better understood in conjunctionwith the following Figures and Detailed Description of their preferredembodiments. The various hardware and software elements used to carryout the inventions are illustrated in the attached drawings in the formof block diagrams and flow charts. For simplicity and brevity, theFigures and Detailed Description do not address in detail features thatare well known in the prior art, such as the literature listed in theBackground of the Invention, above. However, to assure an adequatedisclosure, the specification hereby expressly incorporates by referenceeach and every patent and other publication referenced above in theBackground of the Invention.

FIGS. 1A, 1B and 1C depict side, top and front views, respectively, of aselectable heads-up display worn by a technician, such as a surgeon.

FIG. 2 shows an example for a basic configuration of an integrated headmounted display (HMD) and associated computer system used by atechnician, such as a surgeon.

FIG. 3 is a block diagram of the primary components of the heads-upsurgeon's display of FIG. 2.

FIG. 4 is a more detailed block diagram of a preferred embodiment of theheads-up display system.

FIG. 5 depicts an embodiment for the physical eye-tracking systemimplemented with an infrared laser.

FIG. 6A depicts an embodiment for the display view and associated iconsand menu items as seen by the surgeon wearing the heads-up display on atransparent or see-through screen.

FIG. 6B depicts an embodiment for the display view and associated iconsand menu items as seen by the surgeon wearing the heads-up display, witha patient's vital signs selected for display on a transparent orsee-through screen.

FIG. 6C depicts an embodiment for the display view and associated iconsand menu items as seen by the surgeon wearing the heads-up display, witha view from one of the cameras selected for display on a transparent orsee-through screen.

FIG. 7A depicts an embodiment for the display view and associated iconsand menu items as seen by the surgeon wearing the heads-up display, witha view from one of the cameras selected for display on a portion of thescreen that has been made selectively non-transparent.

FIG. 7B depicts an embodiment for the display view and associated iconsand menu items as seen by the surgeon wearing the heads-up display, withmultiple x-ray images from the computer or one of the cameras selectedfor display on a portion of the screen that has been made selectivelynon-transparent.

FIG. 7C depicts an embodiment for the display view and associated iconsand menu items as seen by the surgeon wearing the heads-up display, withmultiple forms of data and images displayed in windows on a portion ofthe screen.

FIG. 7D depicts another form for the display view and associated iconsand menu items as seen by the surgeon wearing the heads-up display, withmultiple forms of data and images displayed in various windows on aportion of the screen.

FIGS. 8A, 8B and 8C depict various views of an alternative embodimentimplemented with a transparent heads-up display that is not worn by thesurgeon, but rather, is movably stationed over the patient.

DETAILED DESCRIPTION

Shown in FIGS. 1A, 1B and 1C are three views of a head-mounted,selectable, display system 10. In the preferred embodiment shown in theFigures, the display system 10 is worn by a surgeon 12 performing anoperation. However, the system is easily programmed for use by anyindividual performing detailed procedures in which it is advantageous tosee the normal field of view, while also having access to and seeing inthat field of view a variety of forms of data relating to the procedure.Thus, it is envisioned that a display in accordance with this inventionwill have many applications, but is primarily intended for procedureswhere detailed work is performed on relatively stationary objects.Accordingly, while the description below refers repeatedly to the useras a “surgeon”, it should be understood that the other users areincluded in the scope of the invention.

For convenience, the phrases “head-mounted display,” “heads-up display”and “HUD” are used interchangeably throughout this specification. Themajor components of the HUD system 10 as worn by the surgeon 12 includea display screen 14, microphone 16, speaker 17, display driver 18,camera 29, display mirrors 20 and 22, light 31, eye-tracking laser 24,eye-tracking detector 26, and eye-tracking optics 28. In its preferredform, each of these components are integrated into a single, adjustablehead piece that is placed on the surgeon's head 30. Once placed on asurgeon's head 30, the various components are adjusted for properoperation. More specifically, the screen 14 is positioned comfortably infront of and in the normal line of sight of the surgeon's eyes 32, andthe microphone 16 (placed in front of surgical mask 19) and speaker 17are positioned so that the surgeon 12 can communicate with selectedmedical assistants and computers including an electronic speechrecognition system, as discussed in greater detail below. The displaydriver 18 and display mirrors 20 and 22 are adjusted to superimposeselected images and data with light rays 4 from outside of HUD 10 on thedisplay screen 14 via optical path 6. Likewise, the eye-tracking laser24, eye-tracking detector 26, and eye-tracking optics 28 are aligned todetect and communicate eye movement to an eye-tracking computer, as alsodiscussed below. In more compact versions of the head mounted displaysystem discussed above and shown in FIGS. 1A, 1B, 1C, and 2, an eyeglassframe may be used to support the display surface 14, eye-tracking laser24, eye-tracking detector 26, eye-tracking optics 28, or othercomponents as desired.

Referring to FIG. 2, the surgeon 12 wears the HUD system 10 tosimultaneously view selected data and images while performing anoperation on a patient 34. In a preferred form, the surgeon 12 canselectively control the display screen 14 to operate between opaque ortranslucent modes. In the opaque mode, the screen 14 will displayprimarily the data or images generated from a control computer 36 or oneor more video or medical imaging input device(s) 38, for example, whenconducting particularly detailed or internal surgery. In the translucentmode, the surgeon 12 will be able to see through display screen 14 tothe patient 34, while at the same time seeing the data or imagesgenerated by the computer 36 or imaging input device(s) 38.

To simplify the disclosure, only one imaging input device 38 is shown inFIG. 2. However, it is expressly noted that multiple numbers, and anyand all forms, of image-generating systems can be employed in theproposed system. Thus, CCD, video, x-ray, NMR, CAT, and all othermedical imaging systems can be employed. In its most basic form, theimaging input device 38 is mounted on a moveable and computercontrollable manipulator assembly or arm 39, so that it can becontrolled by the surgeon 12 through the HUD system to move to selectedparts of the patient 34, and to obtain and magnify images of particularparts of the body, tissue or organs undergoing surgery. For a moredetailed discussion on automatically controlled and moveable cameras,see U.S. Pat. No. 5,249,045 and the patents cited therein, all of whichare incorporated herein by reference.

In operation, the surgeon 12 can command the HUD system 10, undercontrol of computer 36, to selectively display on screen 14 variousforms of data, graphics or images, including magnified or otherwisemodified images from the imaging input device(s) 38, while at the sametime looking through the screen 14 to view the patient 34 and thenormal, real-life environment of the operating room. Thus, the surgeon12 is able to directly view the patient 34, while at the same time,select from many forms of data for display on the HUD screen 14. Forexample, if the surgeon 12 is performing surgery in close proximity tocritical organs, the surgeon may wish to see both the movements of hisor her hands, and a superimposed magnified view from one of the imageinput device(s) 38. The surgeon 12 can control the video device 38 tomove to and focus on the critical area or surgical site, and to obtainand magnify and image for display on the HUD display 14. In addition,the surgeon can control the computer system 36 to record the generatedimages, and then to display on the HUD screen 14 selected parts thereofafter being magnified or otherwise computer enhanced. In accordance withthe invention, the surgeon 12 has the great advantage of being able tosimultaneously control the HUD and data acquisition systems, while alsolooking through the HUD display 14 to watch minute hand movements andother aspects of the local environment.

As explained in greater detail below, the surgeon 12 may command thecomputer 36 in several ways. In the preferred mode, the HUD system 10incorporates eye-tracking to control a curser that is displayed on theHUD screen 14. More specifically, as shown graphically in FIGS. 6A-6C,the standard display 14 includes menu items or icons that can beselected by a curser 40 that is in turn controlled by an eye-trackingsystem. For example, when the surgeon 12 focuses his eyes on a selectedicon or menu displayed on the HUD screen 14, the eye-tracking systemwill correspondingly cause the curser 40 to move or “track” over theselected icon or menu item. The surgeon can then select the specificicon or menu by voice command or with a foot-operated select button (notshown) operating in a manner similar to the well known “mouse” button.Thus, the surgeon operates eye-tracking curser 40 in a “hands-free”manner to move onto icons and to select various imaging systems,computers or other data sources for display on the HUD display 14.

Alternatively, or in combination with the eye-tracking curser 40, theHUD system 10 includes a standard speech recognition sub-systemintegrated with the command operations. Thus, the surgeon 12 can speakselect speech commands or select words to select specific menus or toinitiate computer operations to acquire and display select images ordata. In the combined speech and eye-tracking mode, the surgeon can usehis or her eye to move the curser 40 to a particular menu or icon, andthen speak commands to perform various operations associatedspecifically with that menu or icon, such as obtaining or magnifyingimages, selecting particular parts of patient histories, etc.

As shown in FIG. 2, the HUD display system 10 preferably communicateswith the command computer 36 via any appropriate form of wirelesscommunication, as is well known in the art of computer networking. Thus,the surgeon is shown wearing a radio transmitter 44 and an antenna 46that operate in a manner that is well known in the art to communicatewith computer 36. Although it is preferred to use wirelesscommunication, it is expressly noted that any and all forms of wirelessor wired communication can be used, and as a result, the invention isnot intended to be limited to any specific form of data communicationbetween the HUD system and the computer 36.

As discussed above, the surgeon may use the HUD system as a hands-freeinterface to control an imaging input device 38 (shown as a camera inFIG. 2) and its associated manipulator assembly 39 to obtain images ofthe patient 34, and to display the images on the HUD display 14. Inaddition, the surgeon can operate the HUD system as an interface toobtain from computer 36 (or from additional remote or local computersnot shown) reference or other patient data, and to display that data onthe HUD display 14.

Shown in FIG. 3 is a general block diagram of the HUD system integratedin a surgeon's environment in accordance with the present invention. TheHUD system 10, including the referenced display, eye-tracking, speechrecognition and communication elements, is coupled to a main commandcontrol computer 36. Also coupled to the command computer 36 are thenumerous sensor inputs 47, such as those that monitor vital signs andacquire medical images. An electronic data base 48 of the patient'shistory is either coupled to or included in the memory of the commandcomputer 36. The command computer 36 has access over a standard network50 to remote sites and computers (not shown).

There are numerous types of sensor inputs 47 that will monitor thepatient 34 and generate data of interest to the surgeon 12. Each suchsensor 47 is considered old in the art, and operates to monitor andgenerate computer data defining the characteristics of the patient 34 ina manner well known to those of ordinary skill in the art. Thus, it isexpressly noted that while several specific types of sensor inputs maybe described in this specification, any and all types of sensor inputs47 can be used, as long as the data is generated in a manner that ismade accessible to the surgeon 12 by the command control computer 36.

The patient data base 48 includes any and all type of patient data thata surgeon may wish to access, including, for example, the patient'shistory and prior medical images. While the data base 48 is shown inFIG. 3 as being separate from the command computer 36, the data base canalso be loaded into the data storage portion of the command computer 36.Likewise, the patient data base 48 may be located remote from thecommand computer 36, and can be accessed over the network 50.

In its preferred form, the network 50 may access not only the standardhospital network, but also remote sites. In that manner, the surgeon 12can access and communicate with other computers, expert systems or datadevices (not shown) that are both local and remote from the surgeon'slocation. Likewise, specialists remote from the specific operating sitemay view the operation and communicate or consult directly with thesurgeon 12. More specifically, the remote sites can selectively displaythe operation from any number of cameras in the operating room, and inaddition, can selectively display and view the procedure with the sameperspective of the surgeon 12 through the HUD display screen 14, usingthe head-mounted camera 29. In that manner, specialists at the remotesites will see what the surgeon 12 sees, including the view of thepatient 34 and the data, menus, icons and cursor shown on the HUDdisplay screen 14. In its preferred form, the video camera 29 is aremote controlled, high-performance camera that is mounted on the HUDgear worn by the surgeon 12 so that it can selectively view and transmitan image of the HUD screen 14 to the command computer 36, and ifdesired, to a remote site or storage device (e.g., disk or video tape)controlled thereby. As shown in FIGS. 1B and 1C, the camera 29 can bemounted on the head of the surgeon 12 in a manner so as to make use ofthe same optics 20 and 22 used by the display driver 18. In addition, asdescribed below, the head mounted camera 29 and/or imaging device 38 maybe mounted instead to the robotic arm 39 controllable by the surgeon totilt, pan, zoom or otherwise focus upon, selected views of the patient.The head mounted camera 29 may also be mounted other than on the top ofthe surgeon's head. For example, the camera 29 can be mounted on theleft side of the surgeon's head, wherein additional optics are used toscan the display screen 14.

FIG. 4 shows a more specific diagram of the main components of the HUDdisplay system 10. The HUD system 10 is coupled to and communicates withthe command control computer 36 via a communication link 52. In itspreferred form, the communications link comprises of two high speeddigital radio transceivers or a optical communication system using fiberoptic cable. The link allows for video, audio, and/or data to betransported to and from a computer network to and from the operator inthe form of graphics, audio, video, text, or other data. For examples ofsuch systems, see R. Gagliardi et al., Optical Communications, (JohnWiley & Sons, Inc. NY, 1995), C. Lynch et al., Packet Radio Networks:Architectures, Protocols, Technologies and Applications, (Pergamon PressNY, 1987), J. Cabral et al., “Multimedia Systems for Telemedicine andTheir Communications Requirements,” IEEE Communications Magazine (July1996), M. Tsiknakis et al., “Intelligent Image Management in aDistributed PACS and Telemedicine Environment,” IEEE CommunicationsMagazine (July 1996), and A. Hutchison, “Electronic Data Interchange forHealth Care,” IEEE Communications Magazine (July 1996). The abovepublications are incorporated herein by reference. It is noted that thecommunication link 52 can use any method of communicating appropriatesignals or information to and from a computer system and/or network(i.e. including but not limited to a radio transceiver, fiber opticcable, wire etc.).

One use of the communication link 52 is to transmit to the commandcontrol computer 36 the input commands 54 generated by the surgeon 12.The surgeon 12 generates the input commands 54 in one or more of severalalternative manners. Primarily, the commands 54 are generated when aneye-tracking system 56 detects the surgeon's eyes 32 focusing onselected icons or menu items displayed on the HUD screen 14. The iconsand menu items then cause the initiation of a corresponding operation,as is common with standard icon-based user interfaces employed withcomputers running the Macintosh or Windows 95 operating systems.Alternatively, the surgeon 12 may generate the commands orally byspeaking select words or commands through microphone 16. A standardvoice recognition sub-system or computer 58 interprets the oral soundsoutput by the microphone 16, and generates digital commands 54 inaccordance with well known speech recognition processes. These speechcommands 54 are then passed to command control computer 36 throughcommunication links 52. For more information on standard speechrecognition systems, see C. Schmandt, Voice Communication WithComputers, (Van Nostrand Reinhold, NY, 1994), and C. Baber et al.,Interactive Speech Technolology: Human Factors Issues in the Applicationof Speech Input/Output to Computers, (Taylor and Francis, Pa., 1993),incorporated herein by reference. For more information on programmingicon or menu based user interfaces, see J. Sullivan et al., IntelligentUser Interfaces, (Addison-Wesley Publishing Company, NY, 1991),incorporated herein by reference.

The communication link 52 is also responsible for routing video imagesfrom a camera and lighting system 49 configured on the HUD system. Morespecifically, the camera and lighting system 49 generate videoinformation under control of the surgeon 12 for display on the HUDscreen 14. Thus, using commands generated by speech or from theicon/menu system, the surgeon controls pan driver 65, tilt driver 67,magnification driver 69 and light 31 to focus upon selected scenes forimaging. The pan driver 65 controls pivotal movement in the horizontaldirection by the camera while the tilt driver 67 controls the verticalpivotal scanning movement of the camera. The magnifier driver 69controls the degree of zoom of the image input device 38. The cameradrivers each control a respective servo-motor, stepper motor or actuatorthat moves or controls the associated camera parameter. In that manner,the surgeon can control the camera to focus upon and scan a particularfeature (such as a tumor), and to generate and display on the HUD screen14 highly magnified views thereof. In addition, the head mounted camera29 can be controlled to scan the HUD screen 14 to generate, record andtransmit to remote sites the view as seen by the surgeon 12. Althoughonly one head mounted camera 29 is actually shown in the drawings, itshould be understood that multiple cameras can be used, includingmultiple different types of cameras (such as video, television,infra-red), and that those and additional cameras may be controlled byother than the surgeon 12. Thus, the imaging input device 38 can eitherbe controlled manually and/or automatically.

In addition to routing input commands 54, eye vector information fromeye-tracking system 56, and data from image input device(s) 38 to thecommand control computer 36, the communication links 52 also serve toroute control information from the command computer 36 to the HUD system10 to operate the various sub-systems such as the speaker 17, displaydriver 18, display screen 14, and imaging input device 38. Morespecifically, the command computer 36 operates to maintain the contentsof the display on HUD screen 14, including maintaining the display ofthe basic menus and icons in accordance with the mode selected by thesurgeon 12, controlling and displaying movement of the cursor 40 (shownin FIGS. 6A, 6B, and 6C) in response to the eye-tracking system 56and/or speech recognition system 58, and displaying data and imagesobtained from the numerous available sources.

Thus, the command computer 36 regularly communicates, throughcommunication links 52, the control signals necessary to operate thedisplay driver or generating system 18, which in turn creates andprojects the required elements of the basic user interface through thedisplay optics 20/22 onto the HUD screen 14. As the surgeon 12 moves hiseyes 32 to focus upon the various icons and menus of the user interface,the eye-tracking system 56 generates input signals for the commandcomputer 36, which in turn controls the display generating system 18 tocorrespondingly move the cursor 40 (shown in FIGS. 6A, 6B, and 6C) onthe display screen 14 in a manner that tracks the movement of thesurgeon's eyes 32. At the same time, the command computer 36 updates thestatus and contents of the various menus and icons, in a manner familiarto those who use a “point-and-click” user interface, such as found incommon Windows '95 and Macintosh computer systems using a mouse,touch-pad or similar device. As various menus and icons are selected,further input signals 54 are generated for use by the command computer36, for example, to obtain patient data or past or current images. Inresponse, the command control computer 36 carries out the requiredoperations external to the HUD system 10 (such as controlling the sensorinputs 47 which may include imaging input device 38 or inputs from database information 48 or other network 50 as shown in FIG. 3) to accessand obtain the requested data. The command computer 36 then controls theHUD system 10 via communication links 52 to update the screen 14 todisplay for the surgeon the requested data or images, and to generateaudio through speaker 17.

The display driver or generating system 18, shown in FIG. 4, operates togenerate the images that are transmitted through the optics 20/22 anddisplayed on the HUD screen 14. Such display drivers or systems are wellknown to the those of ordinary skill in the art, and any applicabledisplay system can be used. For example, it is known to use displaygenerating devices such as CRTs, LEDs, laser diodes, LCDs, etc., andthis invention is not limited to any particular system, as long as itcan generate and display video or computer-generated images onto thedisplay surface/screen 14 or directly into the user's eyes 32 therebysuperimposing images over the surgeon's actual field of view. See, forexample, the following references related to display systems, each ofwhich are incorporated herein by reference: A. Yoshida et al., “Designand Applications of a High-Resolution Insert Head-Mounted-Display”,Proc. VRAIS' 95 (pgs. 84-93, 1995), E. Williams, Liquid Crystals forElectronic Devices (Noyes Data Corporation, NJ, 1975); M. Tidwell etal., “The Virtual Retinal Display—A Retinal Scanning Imaging System,”Proceedings of Virtual Reality World '95 (pgs. 325-334, Munich, Germany:IDG Conferences and Seminars, 1995); J. Kollin, “Optical EngineeringChallenges of the Virtual Retinal Display,” Proceedings of the SPIE(Vol. 2537, pgs. 48-60, Bellingham, Wash., 1995); J. Kollin, “A RetinalDisplay for Virtual-Environment Applications,” Proceedings of Societyfor Information Display (1993 International Symposium, Digest ofTechnical Papers, Vol. XXIV, pg. 827, Playa del Rey, Calif.: Society forInformation Display, 1993) and G. Robinson, “Display Prototype UsesEye's Retina as Screen,” Electronic Engineering Times (pgs. 33-34, Apr.1, 1996).

In its preferred form, the HUD system 10 uses a projection method fordisplaying images in the user's field of view using a light source (e.g.CRT, LED, laser diode, LCD projector, etc.). The light source intensityor brightness can be varied in the user's field of view so that theimage being displayed can be more visible than the surrounding light.The light source intensity may also be decreased so that the surroundinglight can become more visible and the image being displayed lessvisible.

Most CRT, LED, and other projection display methods require distance(optical path length) for projecting images. Thus, as shown in FIG. 1, aCRT/LED projector 18 is used as the display driver. In order to make thedisplay screen 14 as small as possible, display mirrors 20 and 22 areused to transmit the projected images to the screen 14. Morespecifically, mirrors 20 and 22 are located within the head-mountedsystem 10, and are positioned outside of the field of view of the user12. However, they reflect the projected image so that it can besuperimposed over a real scene on the display screen/surface 14 formedof a glass or other suitable display material. Another method ofdisplaying images in the user's field of view is by using LCD technologyembedded inside the display surface 14. Here light from the surroundingenvironment is blocked or filtered by the display when the appropriatevoltage is applied to cells in a LCD matrix. Part of the control of thedisplay screen 14 is done through eye-tracking system 56. Thiseye-tracking system 56 includes eye-tracking electronics 73, an infraredcamera 26, eye-tracking optics 28, and an infrared laser 24, all ofwhich are described below.

Shown in FIG. 5 is an example of an eye-tracking system 56. This systemoperates in a manner known to those of ordinary skill in the art.Several such systems are readily available, and the invention is notlimited to any particular device, system, means, step or method fortracking the eye. For more detail on such eye-tracking systems, see forexample the following U.S. Pat. Nos. 5,231,674; 5,270,748; 5,341,181;5,430,505; 5,367,315; 5,345,281; 5,331,149 and 5,471,542 incorporatedherein by reference. In its preferred form, a low power laser 24generates an infrared eye-tracking laser beam 60. The laser beam isprojected through a lens 62 and reflected by a mirror 28 onto the user'seye(s) 32. The user's eyes include a sclera 64, cornea 66, and pupil 68.When the user's eye(s) 32 move, the eye components cause distortions inthe infrared laser beam, which are reflected back onto mirror 28, andthen through a lens 70 into an infrared photodetector, infrared camera26 or other type of photodetector. This distortion of the laser beamcorresponds to the eye direction vector which can be measured accuratelyby eye-tracking electronics 73 (Shown in FIG. 4). Data defining the eyedirection vector is subsequently transmitted from the eye-trackingelectronics 73 to the command computer 36 through the communicationlinks 52. For calibration, the eye-tracking optics which include mirror28, lens 62, infrared camera 26, and laser 24, may be automaticallyadjusted for optimal performance through the use of computer controlledactuators (not shown).

It is expressly noted that, while separate eye-tracking electronics 73are shown in the block diagram as carried by the heads-up display system10, it is also possible to transmit the raw data from the infrareddetector imaging device 26 to the command computer 36, which thendetermines the associated eye direction vectors. Likewise, theeye-tracking computer (and other electronics, if used) can be worn bythe surgeon 12 on a belt or backpack (not shown).

Shown in FIGS. 6 and 7 are the contents and arrangement of severalpreferred forms of visual displays for the screen 14, includingexemplary icons and menus for the user interface. Referring first toFIG. 6A, an object such a patient 34 is shown visible through thedisplay 14 in the normal field of view 72 of the user 12. Morespecifically, a surgeon 12 wearing the HUD system 10 will see thepatient 34 on table 75 through the semi-transparent display screen 14.The user will also see a number of icons or menu items 74, 76, 78, 80,82, 84 and 86, superimposed over the real scene along the top portion ofthe normal field of view 72. Alternatively, the icons or menu items 74,76, 78, 80, 82, 84, and 86 can be positioned along an opaque portion ofthe display screen 14, outside the normal field of view 72. The specificcontents and form of the icon or menu items 74, 76, 78, 80, 82, 84, and86, along with their associated computer operations, will vary dependingon the specific implementation and programming. However, in itspreferred form, there will be included icons or menu items that allowthe user to control one or more cameras, access one or more computers,control the characteristics of the display 14, and display data, such asa patient's vital signs.

For example, as shown in FIG. 6A, three separate icons or menu items 74,76 and 78, are assigned to control three cameras, indicated as CAM1,CAM2 and CAM3, respectively. By selecting any of these camera icons 74,76, and 78, the user can independently control the associated camerasystems to obtain (pan, tilt, rotate, zoom, etc.) and display variousimages. Similarly, two icons or menu items 80 and 82 are assigned tocontrol access to two computers, indicated as COMP1 and COMP2,respectively. By selecting either of the computer icons, the user canaccess and control the associated computers or networks, to obtainpatient data or other reference material. Another icon or menu item 84is indicated with the label DC and is provided to allow the user toaccess and vary the characteristics of the screen 14. For example, theDC icon or menu item 84 can be accessed by the user to controlbrightness, contrast, and the degree to which you can see through thescreen 14. Another icon 86 allows the user to control various devices toobtain and display on screen 14 any of a wide variety of vital signs.

As discussed above, each of the icons or menu items 74, 76, 78, 80, 82,84, and 86 can be accessed and controlled by causing the eye-trackingcurser 40 to move over and select the desired icon. For example,referring to FIG. 6B, to see an update of the patient's vital signs, thesurgeon can focus his or eyes 32 on the icon 86 corresponding to thepatient's vital signs. The eye-tracking system 56 will track thesurgeon's eyes 32 to cause the cursor 40 to scroll to the VITAL icon 86.Depending on the programming of HUD system 10, when the cursor 40 tracksover the VITAL icon 86, the patient's vital signs will be superimposedover a portion of the surgeon's field of view 72. Shown in FIG. 6B isthe display of the standard vital signs in analog graphic format.Specifically, graphics are shown for the patient's blood pressure 83,heart rate 85, respiratory rate 87 and body temperature 89. However, anyadditional vital sign (e.g., blood sugar, oxygen level, blood flow,etc.) can be programmed into the system and selected by the surgeon fordisplay. In addition to, or in place of, the analog displays 83, 85, 87and 89, digital values and titles can be displayed (not shown).Likewise, the system can be programmed to display the vital signs for aset period of time, continuously, or in an “on-off” fashion.

Referring now to FIG. 6C, there is shown the same view as FIG. 6B, withan image captured by an image input device 38 superimposed on the normalfield of view 72. To select a camera and display its image, the surgeon12 focuses his eyes upon the associated icon, for example, the CAM1 icon74. As above, the eye-tracking system 56 causes the cursor 40 to trackto icon 74, and correspondingly initiates the desired camera image to besuperimposed over the image of the patient 34. In the example shown inFIG. 6C, the image is a 5-times magnified image of an incision 91 in thepatient. If desired, using appropriate menu icons, such as, for example,the display control icon 84, the surgeon may also cause the displayscreen 14 to operate in an opaque mode, displaying only the image of theincision 91 as if on a normal computer display screen. Likewise, thesurgeon can magnify or otherwise control an image input device(s) 38 toobtain the image(s) desired, at the appropriate magnification. Forexample, FIG. 7A shows the display screen 14 operating in an opaque orsemi-transparent mode with the incision 91 magnified to a 10-times view.Also shown in FIG. 7A, the cursor 40 has been replaced with a crosshatch formed by dotted lines 90 and 92, thereby allowing the surgeon 12to precisely select portions of the image to be further still furthermagnified, enhanced, and/or centered in the display. By way of furtherexample, reference is made to FIG. 7B, which depicts the display ofskeletal FIGS. 94 and 96 selected as above by the surgeon 12 moving thecursor 40 to still another of the camera icons, for example, CAM2 icon76.

If desired, the user interface and menu/icon programming can beconfigured to require the surgeon to take further action after thecursor 40 tracks over one of the icons. For example, and in the simplestform, once the surgeon causes the cursor to track over a selected icon,nothing may happen until the surgeon “clicks” a foot-operated mousebutton (not shown). In more complex forms of the invention, the surgeoncan actually access the selected operation by tracking the cursor to theselected icon and then speaking a select code word (e.g., “select” or“open”) into microphone 16, which word or words are interpreted byspeech recognition system 58. In still another form of the invention,the surgeon can access the selected operation associated with aparticular icon or menu item by blinking a set number of times in quicksuccession after tracking the cursor 40 to the desired located. Theblinking action is detected by the eye-tracking system 56.

In its simplest form, the selection by the surgeon of a specific iconwill cause the associated data or images to be displayed on the screen14. In such a mode of operation, it is desirable to include numerousicons on the periphery of the field of view 72, so that the surgeon 12can quickly select an operation. In a more complex form of theinvention, a series of menus can be associated with each icon, each menuhaving successively more detail. For example, instead of having threecamera icons 74, 76 and 78, a single “video” icon can be substituted,which when selected by the cursor 40, will change the display to thenshow the individual icons for each of the many available cameras. Next,when one of the individual camera icons is selected by the cursor 40,the display will again change to show individual icons for the variouscamera controls, such to control the panning, tilting, rotating,magnification, filters, manipulator members, etc.

As indicated, in more complex forms of the invention, the HUD system 10may incorporate a hierarchical icon or menu system, where several layersof menus are necessary to obtain and display desired data. In that case,greater flexibility is desirable in controlling how much data can bedisplayed on the screen 14 at any given time. More specifically, asshown in FIG. 7C, in the more complex forms of the invention, the wellknown programming techniques from the Macintosh or Windows 95 operatingsystems are adapted and included in command computer 36 to allow thedisplay of multiple windows, that can be easily tiled, cascaded,selected, re-sized, or hidden in a manner now familiar to those skilledin the art. Thus, once a specific camera is selected, the surgeon 12 cansimply “hide” the image until it is again desired to view it, at whichpoint it can be easily selected. Thus, the surgeon is not required tosequence through each of the menu levels to access the desired image.For example, as shown in FIG. 7C, the user has configured and controlledthe display to show the windows or regions having patient data 98,magnified camera view 100, MRI data 102, magnified skeletal view 96, andwhole body skeletal view 94. The surgeon 12 can independently hide,re-size, and rearrange each of the windows, along with the transparencylevel of the screen 14, thereby providing a maximum of flexibility.

The flexibility of the system is further shown in FIG. 7D,which depictsthe display 14 having various user selected information, data, or imagespositioned thereon at varying degrees of intensity or transparency. Forexample, in FIG. 7D the surgeon has placed in the center of the display14 a semi-transparent window 108 through which the normal field of view72 displays the patient 34. When the surgeon's eyes 32 are focused onthe window 108, the cursor 40 switches to the eye-tracking cross hairs90 and 92. Under control of the eye-tracking system 56, the cross hairs90 and 92 allow the surgeon 12 to select a specific portion of thepatient 34, such as incision 91. The selected portion (e.g., part of theincision 91) of patient 34 is then locked on and magnified as a separateview in a different window 112 in the manner described above where adifferent set of selectable cross hairs 115 and 117 are shown forfurther magnification. As also shown in FIG. 7D, data such as thepatient name and procedure description is displayed in the title portion114 at the top of the display screen/surface 14. The surgeon 12 hasselected more detailed patient data 116 to be displayed in a separatewindow 116 in the lower left hand comer of display 14. Variousmedication listings 118 and recommendations 120, along with currentcursor or cross-hair coordinates 122 are displayed in a top portion ofscreen 14. Also selected for display are programmable warning indicators124, 126, and 128, generally shown at the top right portion of thedisplay screen/surface 14. The warning indicators may be programmed bythe surgeon 12 to monitor certain conditions and to visually indicatewarnings at desired levels. The same programmable warning indicatorswill issue various levels of audible warning tones to the surgeon 12through the speaker 17. In the configuration of FIG. 7D, the surgeon 12,has selected and displayed the vital signs in a separate window 130 atthe top left corner of the display screen/surface 14, and a 3D model ofa select portion of the patient in window 132, which is continuallyupdated is shown below the vital signs 130 and updated in real time.Other graphical information 110 is updated in real time and displayed atthe bottom right corner of the display screen/surface 14. Otherpertinent images or data may be selectively displayed in other areas ofthe display 14, for example skeletal images in area 131 and magneticresonant imaging (MRI) data 132. All of the described operations canalso be effected or initiated by using the speech recognition system 58.

Thus, the overall medical HUD system 10 is extremely flexible, allowingeach surgeon to customize the display and use only the features deemedappropriate, necessary and desirable. In the simplest operating mode,the surgeon may choose to display on a small part of the screen 14 onlythe patient's vital signs, as shown in FIG. 6B. In other modes, anddepending on the procedure, the surgeon may elect to proceed in asemi-transparent mode, with several windows of data or images, as shownin FIG. 7D.

Each of the variations in programming set forth above can be configuredfor each specific surgeon 12 in a computer file assigned to that userand stored in command computer 36. Thus, if a particular surgeon 12prefers to have specific icons or menus shown on specific screens, orfor example, prefers digital over analog displays for vital signs, thatuser can select those specific settings and the system will perform inaccordance therewith. Similarly, the system allows for substantialcustomization for specific types of surgeons or fields outside ofsurgery (e.g., microelectronics, forensics, etc.).

Shown in FIGS. 8A, 8B, and 8C is still another embodiment of theinvention that incorporates a non-attached heads-up display screen 134.In FIGS. 8A and 8B, the display screen/surface 134 is shown as agenerally flat rectangular semi-transparent display area. The displayscreen/surface 134 is attached to a top end of a sturdy but flexiblepost 136 via joint 138, thereby allowing it to be moved to variousviewable positions. The joint 138 is also constructed to allow thescreen 134 to be moved left or right relative to the post 136. The lowerend of post 136 is attached to a base 140 that is supported on thefloor. The post 136 is also adjustable in height and can be bent toallow precise positioning of the screen/surface 134 by the user. In afurther modification to this non-attached HUD embodiment, as shown inFIG. 8C, the display screen 134 is mounted via robotic arm 142. Displayscreen/surface 134 of HUD 10 is attached at a lower end of themulti-jointed, robotic arm or other manipulator 142. The upper end ofarm 142 is coupled to a mounting platform 144 that is fixed to theceiling 146 of a room. Speech commands such as “Adjust Display On”,“Adjust Display Off”, “Up”, “Down”, “Left”, “Right”, “Forward”, and“Backward” can be used for controlling the position of the retractabledisplay screen/surface 134. Here the position of the displayscreen/surface 134 may be robotically controlled by speech signals froman operator 12. Such speech signals may be received by a microphone andprocessed by a speech recognition system which thereby sends signals toa robotic microcontroller that drives the appropriate actuators toposition the display screen/surface 134 as desired.

In the embodiments of FIGS. 8A, 8B and 8C, the non-attached heads-updisplay system operates in a manner as described in the head-mounted HUDdisplay embodiment of FIGS. 1-7, thereby allowing the surgeon to selectand display data in a superimposed manner over the normal field of viewof the patient 34. Both the speech recognition system 58 andeye-tracking system 56 can be used to move the cursor 40 to activateselect icons or menus for operating computer 36 and display 134.However, in the modified forms of the invention shown in FIGS. 8A, 8Band 8C, additional methods for moving the cursor are possible, includingusing a low power laser mounted on head of the surgeon 12, along with atouch screen incorporated in display screen itself.

The foregoing description of a preferred embodiment and best mode of theinvention known to applicant at the time of filing the application hasbeen presented for the purposes of illustration and description. It isnot intended to be exhaustive or to limit the invention to the preciseform disclosed, and many modifications and variations are possible inthe light of the above teaching. The embodiment was chosen and describedin order to best explain the principles of the invention and itspractical application, and to enable others skilled in the art to bestutilize the invention in various embodiments and with variousmodifications as are suited to the particular use contemplated.

For example, many computer-controlled instruments used in themicro-electronics field have one or more eyepieces through which theoperator views a specimen. The preferred embodiment of the heads-updisplay can be easily modified for use in such applications. Morespecifically, because the user's eye is relatively aligned with theeye-piece, the eye-tracking laser can be unobtrusively mounted in orbelow the eye-piece shaft. The user interface can be displayed by adisplay driver onto the same view seen through the eye-piece. In samemanner as described for the preferred embodiment, the user can see thefull realm of data in the normal field of view, while simultaneouslycontrolling the associated computer. Still further modifications arepossible without departing from the spirit and scope of the invention.

The HUD system's ability to provide high resolution images directly inthe field of view of an operator without forcing the operator to lookaway can greatly enhance the ability to complete an operation in a veryprecise and controlled manner. This precise control can be incorporatedinto surgical cutting, probing, and/or positioning tools by clearlypresenting the position of such tools onto the display with respect to apatient and/or patient model obtained from real time imagery. Thistechnique can be very advantageous in the event that the actualorientation or position of such tool(s) is unobtainable from anunassisted eye but requires direct visual control to operate.

1-55. (canceled)
 56. A method of using a heads-up display system formedical purposes, comprising: (a) supporting from the head of a medicaltechnician components of a computer-controlled heads-up display system,including (i) a projection device, (ii) an eye-tracking device, and(iii) a transparent display screen; (b) positioning the transparentdisplay screen such that the medical technician can see only through thetransparent display screen; (c) activating the projection device toproject onto the transparent display screen a plurality of vital-signicons; (d) tracking eye movement of the medical technician with theeye-tracking device and further activating the projection device toproject onto the transparent display screen a cursor, at a position onthe screen corresponding to the direction of gaze of the medicaltechnician; and (e) generating vital-sign information, concerning apatient who is closest to the medical technician, corresponding to thevital-sign icon at which the eye is focused when (i) the eye of themedical technician is focused on one of the plurality of vital-signicons, and (ii) the computer-controlled heads-up display system detectsthat the medical technician has issued an activation command.